Papers with fusion techniques
Align before Attend: Aligning Visual and Textual Features for Multimodal Hateful Content Detection (2024.eacl-srw)
Copied to clipboard
| Challenge: | Existing approaches to multimodal hateful content detection focus on detecting hate speech from text-based content, but they fail to address modality-specific features. |
| Approach: | They propose a context-aware attention framework for multimodal hateful content detection that integrates an attention layer to meaningfully align the visual and textual features. |
| Outcome: | The proposed framework achieves F1-scores of 69.7% and 70.3% on two hateful meme datasets and shows 2.5% and 3.2% performance improvement over the state-of-the-art systems. |
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph (P18-1)
Copied to clipboard
| Challenge: | Analyzing human multimodal language is emerging area of research in NLP. |
| Approach: | They propose a multimodal fusion technique to exploit how modalities interact in multimodal language. |
| Outcome: | The proposed technique exploits how modalities interact with each other in human multimodal language. |
Extending CLIP’s Image-Text Alignment to Referring Image Segmentation (2024.naacl-long)
Copied to clipboard
| Challenge: | Referring Image Segmentation (RIS) is a cross-modal task that aims to segment an instance described by a natural language expression. |
| Approach: | They propose a framework that leverages the cross-modal nature of CLIP for RIS by leveraging image-text alignment knowledge in CLIP's image-embedding space. |
| Outcome: | The proposed framework outperforms CLIP-based methods on all three major RIS benchmarks and outperformed previous CLIP methods. |
Multi-Channel Spatio-Temporal Transformer for Sign Language Production (2024.lrec-main)
Copied to clipboard
| Challenge: | Sign language production models ignore structural correlations between channels and use multi-channel spatial attention to capture correlations across channels. |
| Approach: | They propose a novel approach to transform sign language into a unified feature representation using multi-channel spatial attention and temporal attention to learn sequential dependencies for each channel over time. |
| Outcome: | The proposed model outperforms state-of-the-art models on two sign language datasets from diverse cultures. |